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1.  Introduction 


The  objective  of  this  research  was  to  experimentally  evaluate  the  effect  of  Computer-Aided 
Robotic  Manipulation  (CARMAN)  and  Computer  Aided  Tele-operation  (CATO)  technologies 
on  manipulator  activity  and  tele-operation,  respectively.  These  technologies  were  assessed  to 
gauge  their  effectiveness  relative  to  standard  operation  of  a  TALON*  TUB.  A  secondary 
objective  was  to  develop  a  set  of  metrics  that  will  test  the  tele-operated  robotic  manipulation  and 
mobility  of  the  CARMAN  and  CATO  technologies  and  may  have  applications  to  other  robotic 
technologies. 

1.1  Background 

The  Battlefield  Automation  Team  (BAT)  of  the  Aviation  and  Missile  Research,  Development 
and  Engineering  Center  (AMRDEC)  Software  Engineering  Directorate  (SED)  has  two  projects 
entitled  CARMAN  and  CATO.  CARMAN  involves  the  development  of  technologies  to  reduce 
robotic  manipulator  task  times  and  improve  the  precision  of  robotic  manipulator  placement. 
CATO  involves  the  development  of  technologies  to  reduce  tele-operated  robotic  mobility  task 
times  and  improve  the  precision  of  tele-operated  driving.  The  first  goal  for  this  experiment  was 
to  operationally  and  technically  measure  CARMAN  and  CATO  technology  performance. 

The  unmanned  systems  community  did  not  have  a  set  of  accepted  metrics  that  measured  robotic 
performance  for  these  tasks,  particularly  the  manipulator  placement  task.  Currently,  measures  of 
effectiveness  for  robotic  systems  include  parameters  such  as  weight,  size,  packability,  usability, 
mean  time  between  failures  (MTBF)  and  mean  time  to  repair  (MTTR).  Although  these  measures 
need  to  be  collected  and  validated  to  ensure  effectiveness  of  the  system,  they  do  not  measure 
operational  performance.  A  second  goal  for  this  study  was  to  develop  a  set  of  robotic  metrics 
that  are  suitable  for  tele-operated  robotic  manipulation  and  mobility,  and  then  to  use  these 
metrics  to  evaluate  the  performance  of  both  CARMAN  and  CATO. 

1.2  CARMAN 

Tele-operated  unmanned  ground  vehicle  (UGV)  manipulators  currently  deployed  in  support  of 
contingency  operations  provide  limited  sensory  data  to  the  operator.  Visual  data  relayed  to  the 
operator  fails  to  provide  accurate  depth  perception  and  an  adequate  field  of  view  while  tactile 
and  auditory  data  are  typically  nonexistent.  The  resulting  shortfall  in  situational  awareness 
fundamentally  limits  manipulator  system  effectiveness  in  terms  of  spatial  precision  and  the  time 
required  to  perform  manipulation  tasks.  Compounding  these  limitations  is  an  inadequate  match 
between  machine  characteristics  and  operator  capabilities.  The  Operator  Control  Units  (OCU) 
supplied  with  current  manipulator  systems  either  burden  the  operator  with  low-level  tasks,  such 
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as  individually  controlling  manipulator  joints,  or  presents  an  overwhelming  set  of  operating, 
modes  designed  for  a  multitude  of  manipulation  tasks.  In  either  case,  current  manipulation 
systems  consume  an  undue  amount  of  an  operator’s  attention,  detracting  from  mission  focus. 

This  in  turn  unnecessarily  lengthens  task  completion  and  operator  exposure  time  in  hostile 
environments  and  increases  the  number  of  assets  needed  to  support  a  given  workload. 

CARMAN  has  two  technologies  that  were  evaluated  in  this  experiment  for  manipulator  control; 
Point  and  Click  and  Fly-To.  The  Point  and  Click  mode  allows  the  operator  to  select  a  point  on 
the  video  from  the  OCU  that  will  direct  the  manipulator  arm  to  move  to  that  general  area.  The 
Fly-To  mode  allows  the  operator  precise  control  over  the  manipulator  arm  via  a  joystick  on  the 
OCU,  to  direct  the  end  effector  to  a  specific  point.  These  modes  are  in  contrast  to  the  current 
manipulator  implementation  strategy,  where  the  operator  actively  moves  the  arm  to  the  location 
via  joystick  control  manipulating  each  joint  movement  independently. 

1.3  CATO 

Soldiers  tele-operate  small  UGVs  in  the  current  force  by  looking  at  a  video  display  or  line  of 
sight  of  the  robot  and  moving  a  joystick.  CATO  was  developed  to  bridge  the  gap  between 
current  and  future  UGVs  where  there  will  be  increased  autonomy.  There  are  four  technologies 
for  CATO;  some  are  only  software,  some  only  hardware,  and  some  a  combination  of  both.  The 
technologies  that  were  developed  were  collaborative  unmanned  aerial  vehicle  (UAV)  driving 
camera,  projected  path  display  driving,  short  distance  operator  selected  waypoint  driving,  and 
elevated  rear  mounted  driving  camera. 

1.3.1  Collaborative  UAV  Driving  Camera 

The  collaborative  UAV  driving  camera  is  teamed  with  a  UAV  and  the  ground  vehicle.  The 
UAV  camera  will  give  an  overhead  visual  perspective  to  the  operator  in  conjunction  with  the 
regular  UGV  driving  camera(s).  This  view  is  also  intended  to  increase  the  view  of  the  area 
surrounding  the  UGV,  increasing  the  operator’s  situational  awareness.  The  UAV  is  positioned 
collaboratively  above  and  slightly  behind  the  UGV.  It  is  commanded  by  the  operator  to  position 
itself  based  on  the  location  and  heading  of  the  UGV.  This  eliminates  the  need  for  the  operator  to 
command  the  UAV.  The  notion  is  that  the  UAV  driving  camera  would  be  added  as  an  additional 
camera  selection  of  the  UGV  so  the  operator  can  select  it  or  any  of  the  other  UGV  cameras  as 
necessary  to  display  current  video.  Due  to  technical  challenges  with  the  UAV  driving  camera 
configuration,  this  CATO  technology  was  not  included  in  this  experiment. 

1.3.2  Projected  Path  Display  Driving 

The  projected  path  display  driving  uses  the  normal  UGV  driving  camera  to  project  a  virtual  path 
on  the  operator  control  display.  The  semi-transparent  path  is  projected  in  front  of  the  actual 
UGV.  The  projected  path  indicates  where  the  UGV  will  travel  if  the  control  input  remains 
constant.  This  technology  was  largely  a  software  solution. 
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1.3.3  Operator- Selected  Waypoint  Driving 

The  short  distance  operator- selected  waypoint  driving  provides  the  operator  the  ability  to 
visually  identify  and  then  select  waypoints  on  the  OCU  display  using  existing  UGV  cameras. 

The  operator  views  the  display  and  identifies  desired  waypoints  using  an  input  device.  The  UGV 
follows  the  waypoints  at  a  selected  speed  and  automatically  stops  at  the  last  specified  waypoint. 
Waypoint  following  allows  the  vehicle  to  move  forward  in  the  battle  without  continuous  operator 
interaction  with  the  robot.  As  a  result  the  operator  can  move  his  position  or  fight  while  using  a 
robot. 

1.3.4  Elevated  Rear-Mounted  Driving  Camera 

The  elevated  rear-mounted  driving  camera  is  a  modification  in  which  the  driving  camera  is 
behind  and  above  the  UGV.  This  perspective  gives  a  third  person  view  of  the  UGV  and 
increases  the  view  of  the  surrounding  area.  This  is  primarily  a  hardware  solution  and  required 
minimal  modification  to  existing  systems. 

The  three  available  CATO  technologies  were  assessed  to  gauge  their  effectiveness  relative  to 
standard  tele-operation  of  a  small  unmanned  ground  vehicle  (SUGV).  CARMAN  and  CATO 
were  implemented  on  the  TALON  IIIB  platform  and  OCU. 

This  report  documents  two  independent  experiments.  The  first  experiment  tested  the  CARMAN 
technology  and  was  conducted  using  a  light  board  and  the  ALTON  (a  modified  TALON  IIIB) 
platform  with  and  without  Point  and  Click  and  Fly-To  technologies.  The  second  experiment  was 
conducted  on  an  obstacle  course  that  was  run  using  the  TALON  IIIB  platform,  with  and  without 
the  CATO  Projected  Path  Display  Driving,  Operator  Selected  Waypoint  Driving,  and  Elevated 
Rear  Driving  Camera  technologies. 


2.  Method 


2.1  Participants 

Participants  were  12  civilian  men  and  women  ranging  in  age  from  18  to  25  years  old,  recruited 
from  the  AMRDEC  workforce.  Participants  had  roughly  the  same  level  of  experience  with 
operation  of  unmanned  systems,  which  was  determined  from  recruiting  techniques  and  verified 
by  the  demographics  questionnaire.  Participants  completed  an  Informed  Consent  form  (appendix 
A)  and  a  Demographics  Questionnaire  (appendix  B)  prior  to  the  beginning  the  experiment. 
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2.2  Apparatus 

2.2.1  Hardware 

2.2. 1.1  The  TALON  IIIB 

The  TALON  IIIB  (figure  1)  is  a  tracked  UGV  that  is  widely  used  for  explosive  ordnance 
disposal  (EOD),  reconnaissance,  communications,  hazmat,  security,  defense  and  rescue 
applications.  The  TALON  weighs  1 15  lb  and  has  a  top  speed  of  5.2  mph.  The  manipulator  arm 
has  three  degrees  of  freedom  and  cannot  rotate  independently  of  the  robot  body.  The  TALON 
IIIB  is  the  primary  robot  used  for  improvised  explosive  device  (IED)  interrogation  by  the  U.S. 
Army  Engineers. 


Figure  1.  TALON  IIIB  and  OCU. 

2.2. 1.2  ALTON 

The  ALTON  robot  is  a  customized  version  of  the  TALON  IIIB  robot  that  has  been  upgraded  to 
improve  sensor  accuracy  and  data  rates.  In  order  to  realize  these  improvements  in  performance, 
it  was  necessary  to  upgrade  the  internal  electronics  and  software  in  both  the  robot  and  the 
operator  control  unit.  The  resulting  increase  in  sensor  accuracy  and  data  rates  allows  smoother, 
more  controlled  motion  of  the  platform  and  its  manipulator.  Although  ALTON  has  new  internal 
electronics,  the  mechanical  chassis  and  motors  are  identical  to  those  of  a  TALON  IIIB. 

2.2. 1.3  OCU 

The  OCU,  shown  in  figure  2,  is  an  encased  control  unit  with  a  display  and  joysticks  that  control 
driving,  manipulator  arm,  and  payloads  (i.e.,  cameras).  The  OCU  used  in  these  experiments 
physically  mirrors  the  external  appearance  of  the  Loster  Miller-TALON  IIIB  OCU  with  the 
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exception  of  a  second  two-axis  controller  and  an  extra  mode  Miller-TALON  IIIB  OCU  with  the 
exception  of  a  second  two-axis  controller  and  an  extra  mode  switch  button  on  the  right  hand  side 
of  the  OCU  face.  Internally,  the  OCU  has  an  onboard  computer  and  custom  electronics  to 
accommodate  the  interface.  The  modifications  to  the  TALON  IIIB  OCU  did  not  affect  how  the 
operator  interacted  with  the  baseline  TALON  IIIB  capabilities.  The  modified  TALON  IIIB  OCU 
with  or  without  the  CARMAN  and  CATO  technologies  is  referred  to  as  ALTON  in  the 
remainder  of  this  document. 


Figure  2.  Operator  control  unit. 

2.2. 1.4  GPS  Data  Logger 

The  data  logger  is  a  custom  data  acquisition  system  that  recorded  GPS  position  data  for  the  UGV 
under  test.  The  data  logger  uses  a  Novatel  GPS  receiver  in  RT2  mode  with  an  accuracy  of  2  cm. 
The  data  logger  is  self-contained  and  powered  by  its  own  battery.  It  is  contained  in  an  aluminum 
enclosure  with  dimensions  (in  inches)  5.125x6.25x3.125,  which  allows  it  to  be  mounted  to  the 
UGV  with  minimal  impact  on  weight  and  volume. 

2.2. 1.5  Light  Board 

A  3x4  ft  rear-projection  light  board  was  used  for  the  CARMAN  tasks  (see  figure  3).  The  Magic- 
Touch  light  board  is  a  positional  measurement  tool  designed  to  report  the  X  and  Y  position  of  an 
object  penetrating  the  area  enclosed  in  the  frame.  The  light  board  is  able  to  determine  the 
position  of  penetration  by  using  an  overlapping  array  of  Infrared  LEDs.  Once  an  object  breaks 
the  beam  it  is  recorded  on  a  computer.  Three-dimensional  (3-D)  perception  is  known  to  be  hard 
for  the  robotic  operator  using  the  standard  TALON  vision  system,  especially  for  novice 
operators.  In  order  to  achieve  “clean”  results,  a  standard  vision  system  was  used  to  minimize  the 
variable  of  assessing  depth,  hopefully  to  reduce  the  variability  that  would  be  induced  by  relying 
on  the  operator's  ability  to  perceive  depth  from  cues  that  are  irrelevant  to  the  technologies  being 
analyzed.  Two  operationally  relevant  actions  were  assessed  in  the  touch  task:  (1)  Pan/Tilt  of  the 
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arm  to  achieve  “bore-sight”  alignment  of  the  gripper  with  the  target  and,  (2)  Maintaining  steady 
trajectory  as  the  gripper  approaches  the  target.  These  two  steps  (along  with  the  use  of  shadows 
to  assess  depth)  are  generally  how  the  robotic  manipulator  is  used  in  real  environments. 
Additionally  the  operator  could  push  through  the  screen  because  it  is  paper,  thus  the  paper  can 
easily  be  repaired/replaced.  By  using  the  Magic-Touch  light  board  the  sensors  registered  a  “hit” 
before  the  pointer  touched  the  paper. 


Figure  3.  Light  board  task. 

2.2.2  Screening  and  Demographic  Measures 

2.2.2. 1  Demographic  and  Computer  Experience  Questionnaire 

The  Demographic  and  Computer  Experience  Questionnaire  (appendix  B)  is  a  12-item 
questionnaire  that  requests  information  regarding  age,  vision,  hearing,  and  computer  experience. 
This  questionnaire  was  used  to  gain  basic  demographic  information  about  the  participant  sample. 

2.2.22  The  NASA-Task  Load  Index  (NASA-TLX) 

Participants  were  given  the  NASA-TLX  subjective  workload  rating  at  the  end  of  each  task 
(appendix  C,  Hart  and  Staveland,  1987). 1  The  NASA-TLX  is  a  multi-dimensional  rating 
procedure  that  derives  an  overall  workload  score  based  on  a  weighted  average  of  ratings  on  six 
subscales  (Mental  Demand,  Physical  Demand,  Temporal  Demand,  Own  Performance,  Effort, 
and  Lrustration).  Scores  range  from  0  (no  workload)  to  100  (extremely  high  workload). 

2. 2. 2. 3  Motion  Symptom  Questionnaire 

This  questionnaire  provides  a  subjective  rating  on  the  participant’s  perceived  levels  of  motion 
sickness  before  and  after  the  experiment  (appendix  D,  Gianaros,  Muth,  Mordkoff,  Levine,  and 


'Hart,  S.  G.;  Staveland,  L.  E.  Development  of  NASA-TLX  (Task  Load  Index):  Results  of  Empirical  and  Theoretical 
Research.  In  P.A.  Hancock  and  N.  Meshkati  Eds.,  Human  Mental  Workload.  Amsterdam:  Elsevier,  1987. 
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Stern,  2001). 2  A  comparison  of  the  pre-  and  post-experimental  response  patterns  across  the 
questionnaire  was  conducted  to  evaluate  motion  sickness.  Any  subjects  showing  elevated  scores 
on  any  of  the  items  was  held  at  the  research  site  until  their  symptoms  abated.  No  motion 
sickness  was  reported  during  the  experiments. 

2.2.3  General  Procedures 

Prior  to  the  start  of  the  experiment,  the  experimenter  briefed  the  participants  on  the  purpose  and 
procedures  of  the  experiment.  Participants  who  agreed  to  take  part  in  the  study  signed  the 
Volunteer  Agreement  Affidavit  (appendix  A)  and  were  given  the  required  briefing  regarding 
confidentiality  as  indicated  on  DA  Form  5303-R.  In  anticipation  of  possible  concerns  regarding 
personal  answers  on  some  of  the  questionnaires,  the  investigators  also  described  the  deliberate 
actions  taken  when  handling  research  data.  In  order  to  ensure  that  individual  data  was  not 
reported  or  revealed  to  anyone,  each  form  was  reviewed  upon  receipt  by  one  of  the  investigators. 
If  any  identifying  information  appeared  on  the  questionnaires  (such  as  name,  social  security 
number,  birth  date,  etc.),  the  investigators  deleted  the  identifying  information  and  replaced  it 
with  a  neutral  code  number.  This  code  number  became  the  participant  identification  number 
used  in  data  files. 

Participants  were  told  that  they  would  complete  two  experiments  in  an  8  h  block,  over  the  course 
of  one  day.  In  experiment  I,  participants  manipulated  the  ALTON  manipulator  arm  with  and 
without  the  CARMAN  Point  and  Click  and  Fly-To  technologies.  In  experiment  II,  the 
participant  tele-operated  the  TALON  IIIB  through  an  outdoor  obstacle  course  with  and  without 
the  CATO  Elevated  Rear-Mounted  Driving  Camera  Mode,  Projected  Path  Display  Driving 
Mode,  and  Operator-Selected  Waypoint  Driving  Mode  technologies.  There  was  a  30  min  break 
between  each  experiment  and  45  min  given  for  lunch.  One-half  of  the  participants  completed 
experiment  I  and  then  experiment  II.  The  other  half  completed  experiment  II  first  followed  by 
experiment  I. 

2.3  Experiment  I — CARMAN  Procedures 

After  a  description  of  the  purpose,  participants  were  given  an  overview  of  and  training  on  the 
baseline  configuration  of  the  CARMAN  configurations.  The  experimenter  presented  the 
functionality  of  the  CARMAN  OCU.  The  participant  completed  a  training  trial.  They  had  the 
opportunity  to  repeat  the  training  run  again  until  they  were  comfortable  controlling  the  robot. 
Criteria  for  efficient  training  were  determined  by  a  subject  matter  expert  on  the  platform  who 
was  present  during  the  training  mission.  These  subject  matter  experts  were  the  engineers  of  the 
CARMAN  and  CATO  software  and  they  used  their  knowledge  of  the  system  to  assess  training 
effectiveness.  The  objective  of  the  CARMAN  task  was  to  manipulate  the  manipulator  arm  to  hit 
20  targets  that  were  displayed  on  the  rear-projection  light  board.  The  participants  were 
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instructed  to  perform  this  task  as  quickly  and  accurately  as  possible.  Figure  4  shows  the 
manipulator  pointing  at  a  target  on  the  light  board. 


Figure  4.  Manipulator  pointing  at  a  target  on  the  light  board  task. 


All  targets  were  the  same  size  bull’s-eye- shaped  figures  that  were  projected  in  three  random 
patterns  on  the  light  board.  Each  pattern  contained  20  targets  that  were  evenly  distributed 
throughout  the  target  board,  meaning  that  in  each  pattern,  special  care  was  taken  to  ensure  that 
equal  numbers  of  quadrants  were  used  and  that  the  space  between  targets  was  similar. 
Participants  completed  two  trials  per  configuration,  baseline  configuration  (i.e.,  ALTON), 
CARMAN  Point  and  Click,  and  Fly-To  configuration.  Participants  were  instructed  to  use  the 
manipulator  arm  to  touch  20  targets  that  appeared  in  succession  on  the  light  board  as  quickly  and 
accurately  as  possible.  Each  target  appeared  and  stayed  lit  until  the  participant  touched  the 
manipulator  end  point  somewhere  in  the  target  or  the  time  for  a  target  hit  ran  out  (20  s).  If  the 
manipulator  arm  touched  some  place  other  than  inside  the  target,  it  was  considered  an 
inadvertent  contact.  Accuracy  was  measured  by  distance  from  the  center  of  the  bull’s-eye.  Once 
a  hit  occurred  the  target  disappeared  and  the  next  target  appeared.  The  participant  continued 
until  all  20  targets  were  hit.  Each  participant  completed  six  trials  using  the  ALTON  OCU  (see 
figure  5). 
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Figure  5.  Participant  controlling  the  manipulator  arm  through  the  operator 
control  unit. 

Participants  completed  the  task  with  only  visual  sightings  of  the  target  board  through  the  OCU. 
A  tarp  separated  the  participant  from  the  TALON  platform  rendering  the  task  as  non-line-of- 
sight.  Additionally,  participants  could  not  see  the  entire  target  board  through  the  OCU.  As  a 
result,  participants  often  had  to  back  up  the  manipulator  arm  to  get  a  wider  view  of  the  target 
board,  which  allowed  them  to  identify  the  target  area.  Speed  and  accuracy  were  stressed. 
Subject  matter  experts  observed  the  trials  for  technical  difficulties.  After  each  CARMAN  task 
trial,  the  participants  completed  the  NASA-TLX  and  were  given  the  opportunity  to  take  a  thirty- 
minute  break  before  beginning  the  next  experiment. 

2.3.1  Experiment  I — Experimental  Design 

The  experimental  design  of  experiment  I  was  a  3x2  within  subjects  design.  There  were  two 
independent  variables,  configuration  type  with  three  levels  (baseline  [i.e.,  ALTON],  Point  and 
Click,  and  Fly-To)  and  trial  with  two  levels  (trial  1  and  trial  2).  Thus,  the  participant  completed 
two  trials  in  the  baseline  configuration,  two  in  Point  and  Click,  and  the  two  in  the  Fly-To 
configuration.  In  all,  each  participant  completed  six  trials.  In  each  trial  there  were  20  targets  to 
touch  with  the  manipulator  on  the  light  board.  Task  completion  time  for  experiment  I  was 
approximately  2  h. 
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Configuration  Type: 


•  A=Baseline  (ALTON) 

•  B=CARMAN  with  Point  and  Click 

•  C=C ARMAN  with  Fly-To 

Repetition: 

•  1  and  2 

Target  Order: 

•  (1),  (2),  (3) 

Order  of  the  conditions  was  counterbalanced  using  a  Williams  Square  Design  (see  table  1). 
Table  1.  Older  of  conditions  for  experiment  I. 


Participant 

Trial  1 

Trial  2 

Trial  3 

Trial  4 

Trial  5 

Trial  6 

1 

A  1(2) 

Bl(3) 

0(1) 

A2(3) 

B2(l) 

C2(2) 

2 

B  1(1) 

Cl  (2) 

Al(3) 

B2(2) 

C2(3) 

A2(l) 

3 

Cl  (3) 

Al(l) 

Bl(2) 

C2(l) 

A2(2) 

B2(3) 

4 

Al(l) 

Cl  (2) 

B  1(3) 

A2(2) 

C2(3) 

B2(l) 

5 

Bl(2) 

Al(3) 

0(1) 

B2(3) 

A2(l) 

C2(2) 

6 

Cl  (3) 

Bl(l) 

A  1(2) 

C2(l) 

B2(2) 

A2(3) 

7 

Al(l) 

Bl(2) 

0(3) 

A2(2) 

B2(3) 

C2(l) 

8 

Bl(2) 

0(3) 

Al(l) 

B2(3) 

C2(l) 

A2(2) 

9 

Cl  (3) 

Al(l) 

Bl(2) 

C2(l) 

A2(2) 

B2(3) 

10 

Al(l) 

0(2) 

Bl(3) 

A2(2) 

C2(3) 

B2(l) 

11 

Bl(2) 

Al(3) 

0(1) 

B2(3) 

A2(l) 

C2(2) 

12 

Cl  (3) 

Bl(l) 

A  1(2) 

C2(l) 

B2(2) 

A2(3) 

Dependent  Variables: 

The  following  dependent  variables  were  measured  by  automatic  data  collection  and  survey 
instruments  during  experiment  I. 

Time  to  complete  (milliseconds)  (digitally  recorded  with  embedded  automatic  data  collection 
software): 

•  Average  Time  per  Trial 

•  Average  Time  per  Target 

Precision  and  Errors  frequency  (digitally  recorded  with  embedded  automatic  data  collection 
software): 

•  Inadvertent  Contacts 

•  Missed  Targets  (Timed  out) 
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Subjective  Responses  (surveys): 

•  Workload  was  measured  with  the  NASA-TLX. 

Analyses  for  experiment  I: 

•  Means  and  standard  errors  were  calculated  for  each  dependent  variable. 

To  examine  the  effects  of  CARMAN  configuration  type  on  operator  performance  (e.g.,  time, 
precision),  a  mixed  linear  model  analysis  was  conducted.  Subsequent  pairwise  comparisons 
were  conducted  for  the  significant  dependent  variable(s).  Effects  showing  significance  of 
p<0.05  were  considered  statistically  significant. 

To  examine  the  effects  of  CARMAN  configuration  type  on  workload,  a  MANOVA  was 
conducted  with  scores  on  the  six  subscales  of  the  NASA-TLX  (Mental  Demand,  Physical 
Demand,  Temporal  Demand,  Own  Performance,  Effort,  and  Frustration)  as  the  dependent 
variables.  Effects  showing  significance  of  p<0.05  were  considered  statistically  significant. 
Subsequent  repeated  measures  ANOVAs  were  conducted  for  the  significant  dependent 
variable(s). 

2.4  Experiment  II — CATO  Procedures 

After  a  description  of  the  purpose,  participants  were  given  an  overview  of  the  CATO  and 
training  on  the  baseline  CATO  configuration.  The  experimenter  presented  the  functionality  of 
the  OCU  for  CATO.  The  participant  completed  a  training  trial.  Criteria  for  efficient  training 
were  determined  by  a  subject  matter  expert  on  the  platform  who  was  present  during  the  training 
mission.  These  subject  matter  experts  were  the  engineers  of  the  CARMAN  and  CATO  software 
and  used  their  knowledge  of  the  system  to  assess  training  effectiveness.  After  training,  the 
participant  completed  the  CATO  tasks,  the  objective  was  to  maneuver  the  unmanned  system 
through  the  obstacle  course. 

The  obstacle  course  consisted  of  two  segments:  path  following  and  obstacle  negotiation.  The 
path  following  segment  consisted  of  following  a  painted  orange  line  through  seven  corners  and 
curves.  The  obstacle  segment  consisted  of  traveling  through  four  gates  (2  widthsx2  approach 
orientations),  a  ramp,  a  narrow  passageway,  a  tightly- spaced  five  cone  slalom,  a  gate  on  a  curve, 
and  a  four  cone  slalom  (a  total  of  16  obstacles).  For  each  trial,  the  operator  navigated  the  course 
in  one  continuous  run,  with  individual  segment  times  being  recorded  by  stopwatch.  Collisions 
with  obstacles  were  noted  by  a  data  collector.  All  tele-operation  was  conducted  “non-line-of- 
sight”  from  a  nearby  climate  controlled  shelter;  this  means  that  at  no  time  was  the  participant 
able  to  directly  view  the  robot.  Participants  were  instructed  to  stress  accuracy  (precision  path 
following  and  avoiding  obstacle  collisions),  but  to  also  drive  as  quickly  as  they  felt  they  could 
while  remaining  accurate  in  their  maneuvers.  Subject  matter  experts  observed  the  operators’  use 
of  the  equipment  during  the  trials  for  usability  issues.  After  each  CATO  task  trial,  the  participant 
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completed  the  NASA-TLX  and  was  given  the  opportunity  to  take  a  30-min  break  before 
beginning  the  next  experiment.  The  driving  course  is  shown  in  figures  6-8. 


Figure  6.  Sketch  of  CATO  driving  course. 


Figure  7.  TALON  on  path  following  course. 
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Figure  8.  ALTON  negotiating  obstacles. 

2.4.1  Experimental  Design 

The  experimental  design  of  experiment  II  was  a  4x2  within  subjects  design.  There  were  two 
independent  variables,  configuration  type  with  four  levels  (baseline  [i.e.,  TALON],  Elevated 
Rear  Mounted  Driving  Camera,  Projected  Path  Display  Driving,  and  Operator  Selected 
Waypoint  Driving)  and  repetition  with  two  levels  (trial  1  and  trial  2).  Thus,  the  participant 
completed  two  trials  in  the  baseline  configuration,  two  in  the  CATO  Elevated  Rear  Mounted 
Driving  Camera  configuration,  two  in  the  CATO  Projected  Path  Display  Driving  configuration, 
and  two  in  the  CATO  Operator  Selected  Waypoint  Driving  configuration.  In  all,  each  participant 
completed  eight  trials.  The  obstacle  course  remained  in  a  static  configuration,  trials  alternated 
running  the  course  from  “start  to  finish”  and  from  “finish  to  start”  to  maintain  consistent  course 
length  but  reduce  learning  effect.  Task  completion  time  for  experiment  II  was  approximately 
3  h. 

Configuration  Type: 

•  A=Baseline  (ALTON) 

•  B=CATO  with  Elevated  Rear  Mounted  Driving  Camera 

•  C=CATO  with  Projected  Path  Display  Driving 

•  D=CATO  with  Operator  Selected  Waypoint  Driving 
Trial: 


•  1  and  2 
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Order  of  the  conditions  was  counterbalanced  using  a  Williams  Square  Design  as  shown  in  table 

2. 


Table  2.  Order  of  conditions  for  experiment  II. 


Participant 

Trial  1 

Trial  2 

Trial  3 

Trial  4 

Trial  5 

Trial  6 

Trial  7 

Trial  8 

1 

A1 

D1 

B1 

Cl 

A2 

D2 

B2 

C2 

2 

B1 

A1 

Cl 

D1 

B2 

A2 

C2 

D2 

3 

Cl 

B1 

D1 

A1 

C2 

B2 

D2 

A2 

4 

D1 

Cl 

A1 

B1 

D2 

C2 

A2 

B2 

5 

Cl 

B1 

D1 

A1 

C2 

B2 

D2 

A2 

6 

D1 

Cl 

A1 

B1 

D2 

C2 

A2 

B2 

7 

A1 

D1 

B1 

Cl 

A2 

D2 

B2 

C2 

8 

B1 

A1 

Cl 

D1 

B2 

A2 

C2 

D2 

9 

A1 

B1 

Cl 

D1 

A2 

B2 

C2 

D2 

10 

B1 

D1 

A1 

Cl 

B2 

D2 

A2 

C2 

11 

Cl 

A1 

D1 

B1 

C2 

A2 

D2 

B2 

12 

D1 

Cl 

B1 

A1 

D2 

C2 

B2 

A2 

Dependent  Variables: 

The  following  dependent  variables  were  measured  during  experiment  II. 

Time  in  seconds  (recorded  by  the  U.S.  Army  Research  Laboratory  [ARL]  data  collector  with 
stopwatch): 

•  Time  to  complete  path  following  segment 

•  Time  to  complete  obstacle  segment 
Precision  and  Errors: 

•  Path  following  precision  (measured  by  GPS  data  logger) 

•  Number  of  obstacle  collisions  (measured  by  ARL  data  collector) 

Power  consumption  used  by  the  Robot  (recorded  by  ARL  data  collector  using  in-line  power 
consumption  meter) 

Subjective  Responses  (surveys): 

•  Workload  was  measured  with  the  NASA-TLX. 

Analyses  for  experiment  II: 

•  Means  and  standard  errors  were  calculated  for  each  dependent  variable. 

Path  following  precision  was  evaluated  by  calculating  the  area  of  deviation  between  the  path 
taken  and  the  true  course.  The  output  is  a  deviation  score  in  pixels  determined  by  the  program 
that  calculates  the  area  between  the  lines.  This  method  was  chosen  for  its  simplicity  and 
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insensitivity  to  path  following  time.  To  examine  the  effects  of  CATO  configuration  type  on 
operator  performance  (e.g.,  time,  precision),  a  mixed  linear  model  analysis  was  conducted. 
Subsequent  pairwise  comparisons  were  conducted  for  the  significant  dependent  variable(s). 
Effects  showing  significance  of  p<0.05  was  considered  statistically  significant. 

To  examine  the  effects  of  CATO  configuration  type  on  workload,  a  MANOVA  was  conducted 
with  scores  on  the  six  subscales  of  the  NASA-TLX  (Mental  Demand,  Physical  Demand, 
Temporal  Demand,  Own  Performance,  Effort,  and  Frustration)  as  the  dependent  variables. 
Effects  showing  significance  of  p<0.05  was  considered  statistically  significant.  Subsequent 
repeated  measures  ANOVAs  were  conducted  for  the  significant  dependent  variable(s). 


3.  Results 


3.1  CARMAN  Results 

To  examine  the  effects  of  the  three  CARMAN  configurations  (ALTON,  Point  and  Click,  and 
Fly-To)  and  Repetition  (1  and  2)  on  performance  in  the  light  board  task,  repeated  measures 
ANOVAs  and  subsequent  pairwise  comparisons  were  conducted. 

Figure  9  presents  a  graph  of  inadvertent  contacts  on  the  light  board  task.  Results  for  the  light 
board  task  showed  that  the  number  of  inadvertent  contacts  was  highest  in  the  ALTON 
configuration.  There  was  an  80%  and  43%  average  reduction  in  inadvertent  contacts  with  the 
Fly-To  and  Point  and  the  Click  configurations,  respectively. 


Note:  *  =  a  significant  difference. 

Figure  9.  Graph  of  the  percentage  of  inadvertent  contacts  for  CARMAN  configurations. 
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A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CARMAN  configuration  for 
inadvertent  contacts,  F  (2,22)  =  1 8.01 ,  /;<(). 01 .  There  was  no  difference  in  missed  targets 
between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,11)  =  0.04,  p< 0.83 
and  F  (2,22)  =  0.44,  p<0.64,  respectively.  To  explain  the  main  effect  of  configuration  on  missed 
targets,  pairwise  comparisons  were  conducted.  Results  showed  that  significantly  more  targets 
were  missed  in  the  ALTON  than  the  Point  and  Click  or  Fly-To  configurations,  p<0.01  for  each. 
Furthermore,  more  targets  were  missed  in  the  Point  and  Click  than  the  Fly-To  configuration, 

p<0.01. 

Figure  10  presents  a  graph  of  Mean  (Standard  Error  of  the  Mean  [error  bars])  Missed  Targets  for 
CARMAN  configurations  on  the  light  board  task.  Results  for  the  light  board  task  showed  that 
the  number  of  misses  was  higher  in  the  Point  and  Click  configuration  than  in  the  Baseline 
(ALTON)  configuration.  The  number  of  misses  in  the  Fly-To  configuration  was  less  than  a  third 
of  the  number  of  misses  in  the  Baseline  (ALTON)  configuration. 


Figure  10.  Graph  of  mean  (standard  error  of  the  mean)  missed  targets  for  CARMAN 

configurations;  percent  change  relative  to  baseline  (ALTON)  in  missed  targets 
is  represented  above  each  bar. 
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A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CARMAN  configuration  for 
number  of  missed  targets  (out  of  twenty),  F  (2,  22)  =  4.35,  p<0.02.  There  was  no  difference  in 
missed  targets  between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,1 1)  = 
3.98,  p< 0.07  and  F  (2,22)  =  2.46,  p<0.10,  respectively.  To  explain  the  main  effect  of 
configuration  on  missed  targets,  pairwise  comparisons  were  conducted.  Results  showed  that 
more  targets  were  missed  in  the  Point  and  Click  than  the  Fly-To  configurations,  p<0.01.  No 
other  comparisons  were  statistically  significant.  Although  there  was  a  large  difference  between 
the  Fly-To  configuration  and  Baseline,  this  difference  was  not  statistically  significant.  It  is 
important  to  note  that  an  87.5%  reduction  in  missed  targets  is  notable  and  has  practical 
significance. 

Figure  1 1  presents  a  graph  of  mean  task  completion  time  (seconds)  on  the  light  board  task. 
Results  for  the  light  board  task  showed  that  task  time  was  shortest  in  the  Fly-To  mode. 
Furthermore,  task  time  was  slightly  shorter  in  the  ALTON  configuration  than  the  Point  and  the 
Click  configuration. 


Note:  *=  p<0.01. 


Figure  11.  Graph  of  mean  (standard  error  of  the  mean)  time  per  target  for  CARMAN  configurations. 


A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CARMAN  configuration  for 
task  completion  time,  F  (2,22)  =  93.79,  /;<().  01.  There  was  no  significant  difference  in  task 
completion  time  between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,1 1) 

=  2.96,  p<0. 10  and  F  (2,22)  =  0.448,  p<0.64,  respectively.  To  explain  the  main  effect  of 
configuration  on  task  time,  pairwise  comparisons  were  conducted.  Results  showed  that  task  time 
was  significantly  shorter  in  the  Fly-To  configuration  than  the  Point  and  Click  or  the  ALTON 
configuration,  p<0.01  for  each.  Task  time  was  significantly  shorter  in  the  ALTON  than  in  the 
Point  and  Click  configuration,  /;<(). 01 . 
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Figure  12  presents  a  graph  of  mean  total  time  to  complete  the  light  board  task.  Results  for  the 
light  board  task  showed  that  the  Point  and  Click  configuration  took  the  longest  to  complete.  The 
Fly-To  configuration  had  the  shortest  task  completion  time. 


Note:  *=p<0.01. 


Figure  12.  Graph  of  mean  (standard  error  of  the  mean)  total  task  time  (seconds)  for 
CARMAN  configurations. 

A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CARMAN  configuration  for 
total  task  time,  F  (2,  22)  =  92.43,  /?<().()  1 .  There  was  no  difference  in  time  between  trial  1  and 
trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,11)  =  3.52,  /;<(). 08  and  F  (2,22)  =  0.37, 
p< 0.69,  respectively.  To  explain  the  main  effect  of  configuration  on  total  task  time,  pairwise 
comparisons  were  conducted.  Results  showed  that  task  time  was  significantly  longer  in  the 
ALTON  and  Point  and  Click  configurations  than  the  Fly-To  configuration,  p<0.01  for  each. 
Furthermore,  task  time  was  significantly  longer  in  the  Point  and  Click  than  the  ALTON,  p<0.01. 

To  examine  the  effects  of  the  three  CARMAN  configurations  (ALTON,  Point  and  Click,  and 
Fly-To)  on  subjective  workload,  a  Multivariate  Analysis  of  Variance  (MANOVA)  was 
conducted.  Subsequent  ANOVAs  and  pairwise  comparisons  were  conducted  for  the  significant 
workload  subscales. 
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Figure  13  presents  a  graph  of  mean  workload  across  the  CARMAN  configurations.  Results 
showed  that  subjective  workload,  specifically  mental  demand,  physical  demand,  temporal 
demand,  and  frustration,  were  highest  in  the  ALTON.  Furthermore,  workload  on  those  subscales 
was  lowest  in  the  Fly-To  configuration  (also  shown  by  percentages  in  table  3). 


Figure  13.  Graph  of  mean  (standard  error  of  the  mean)  subjective  workload  for  CARMAN 
configurations. 

Table  3.  Table  of  percentages  of  workload  from  baseline  (ALTON)  by  configuration. 


Mental 

Physical 

Temporal 

Performance 

Effort 

Frustration 

Overall 

Workload 

Baseline 

(ALTON) 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

Fly-To 

-29.6%a 

-39.8 a 

-22.0%a 

-26.0% 

-5.2% 

-47.1%a 

—25.  l%a 

Point  and  Click 

-29.6%a 

-38.5%a 

-11.8% 

-9.8% 

-3.5% 

-15.7%a 

-16.2%a 

‘‘Significant  comparison  between  configuration  and  baseline  (ALTON),  p< 0.05. 


A  repeated  measures  MANOVA  revealed  a  significant  main  effect  of  CARMAN  configuration 
on  workload,  F  (12,  32)  =  2.92,  /;<.()  1 .  There  was  no  difference  in  workload  between  trial  1  and 
trial  2  or  an  interaction  of  trial  with  configuration,  F  (6,5)  =  1.10,  p  =  0.46  and  F  (12,32)  =  1.04, 
p  =  0.43,  respectively.  Subsequent  ANOVAs  were  conducted  to  determine  which  subscales  of 
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the  NASA-TLX  workload  scale  contributed  to  this  main  effect.  There  were  significant  main 
effects  for  mental  demand,  physical  demand,  and  frustration,  F  (2,20)  =  9.65,  p< 0.01,  F  (2,20)  = 
7.53,  p<0.01,  and  F  (2,20)  =  5.37,  p<0.01,  respectively.  Temporal  demand  was  marginally 
significant,  F  (2,20)  =  3.24,  p< 0.06.  No  other  subscales  were  significant.  To  explain  the  main 
effect  of  configuration  on  the  significant  subscales,  pairwise  comparisons  were  conducted. 
Results  showed  that  mental  demand  was  significantly  higher  in  ALTON  than  Fly-To  or  Point 
and  Click  configurations.  Physical  demand  was  significantly  higher  in  the  ALTON  than  the  Fly- 
To  or  Point  and  Click  configurations,  p<0.02  and  p< 0.01,  respectively.  Temporal  demand  was 
significantly  higher  in  the  ALTON  configuration  than  the  Fly-To  configuration,  p< 0.01. 
Frustration  was  significantly  higher  in  the  ALTON  and  Point  and  Click  configurations  than  the 
Fly-To  configuration,  /;<().  03  and  p< 0.01,  respectively.  Although  performance  was  not 
statistically  significant,  the  differences  between  configurations  were  practically  significant. 
Perceived  performance  was  highest  in  Fly-To  than  in  all  other  configurations. 

A  second  repeated  measures  ANOVA  was  conducted  for  overall  workload.  There  was 
significant  main  effect  of  CARMAN  configurations  for  overall  workload,  F  (2,20)  =  7.65, 
p  <0.01.  There  was  no  difference  in  workload  between  trial  1  and  trial  2  or  an  interaction  of  trial 
with  configuration,  F  (1,10)  =  2.63,  p  <0.13  and  F  (2,20)  =  0.09,  p  =  0.91,  respectively.  To 
explain  the  main  effect  of  configuration  on  overall  workload,  pairwise  comparisons  were 
conducted.  Results  showed  that  overall  workload  was  significantly  higher  in  the  ALTON  than 
the  Fly-To  or  Point  and  Click  configurations,  /xO.Ol. 

3.2  CATO  Results 

Repeated  measures  ANOVA  and  pairwise  comparisons  were  used  to  analyze  the  performance 
differences  of  the  four  CATO  configurations  (Baseline  (TALON)),  Projected  Path,  Elevated 
Camera,  and  Waypoint) 

Figure  14  presents  the  mean  path  following  time  (seconds)  for  the  four  CATO  configurations. 
The  percent  change  from  Baseline  (TALON)  is  listed  over  each  bar.  Significant  differences 
(p’ s<0.05)  from  Baseline  are  marked  with  an  asterisk.  For  the  path  following  course,  Baseline 
was  the  quickest  while  Waypoint  mode  was  the  slowest. 

A  mixed  linear  model  analysis  revealed  a  significant  main  effect  of  CATO  configuration  for  path 
following  time,  F  (3,33)  =  24.02,  p< 0.01.  There  was  a  significant  effect  in  path  following  time 
between  trial  1  and  trial  2,  F  (1,11)  =  5.1 1,  p<0.05,  but  there  was  no  interaction  of  trial  with 
configuration,  F  (3,33)  =  0.99,  p<0.40.  To  explain  the  main  effect  of  configuration  on  path 
following  time,  pairwise  comparisons  were  conducted.  Results  showed  that  significantly  more 
time  was  needed  to  complete  the  path  following  portion  for  the  Waypoint  configuration  than  the 
other  three  configurations,  /?<0.0 1  for  each.  There  were  no  significant  differences  between  the 
other  three  modes. 
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Note:  *  =  a  significant  difference. 


Figure  14.  Graph  of  mean  (standard  error  of  the  mean)  path  following  completion  time  for 
CATO  configurations. 

Figure  15  presents  the  mean  path  deviation  (pixels)  for  the  four  CATO  configurations.  The 
percent  change  from  Baseline  (TALON)  is  listed  over  each  bar.  Significant  differences  (/;<(). 05) 
from  Baseline  are  marked  with  an  asterisk.  Projected  Path  was  the  most  precise  at  path 
following  and  Waypoint  was  the  least  precise. 


Note:  *  =  a  significant  difference. 

Figure  15.  Graph  of  mean  (standard  error  of  the  mean)  path  following  deviation 
for  CATO  configurations. 
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A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CATO  configuration  for  path 
following  deviation,  F  (3,33)  =  9.81,  p<0.01.  There  was  no  significant  effect  in  path  following 
deviation  between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,1 1)  =  0.44, 
p< 0.52  and  F  (3,33)  =  1.28,  p< 0.29,  respectively.  To  explain  the  main  effect  of  configuration  on 
path  following  deviation,  pairwise  comparisons  were  conducted.  Results  showed  that  Projected 
Path  configuration  was  significantly  more  precise  than  the  other  three  configurations,  p’s<0.01. 
Results  also  showed  that  the  Waypoint  configuration  was  significantly  less  precise  than  the  other 
three  configurations,  p’s<0.03.  There  was  no  significant  difference  between  the  Baseline  and 
Elevated  Camera  configuration. 

Figure  16  presents  the  mean  obstacle  course  completion  time  (seconds)  for  the  four  CATO 
configurations.  The  percent  change  from  Baseline  (TALON)  is  listed  over  each  bar.  Significant 
differences  (p’s<0.05)  from  Baseline  are  marked  with  an  asterisk.  Elevated  Camera  was  the 
fastest  mode  for  this  course  and  Waypoint  was  the  slowest. 


Note:  *  =  a  significant  difference. 

Figure  16.  Graph  of  mean  (standard  error  of  the  mean)  obstacle  course  completion  time  for  CATO  configurations. 
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A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CATO  configuration  for 
obstacle  course  time,  F  (3,33)  =  52.15,  /;<(). 01 .  There  was  no  significant  effect  in  obstacle 
course  time  between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,1 1) 

=  4.33,  p<0.062  and  F  (3,33)  =  0.95,  p<0.42,  respectively.  To  explain  the  main  effect  of 
configuration  on  obstacle  course  time,  pairwise  comparisons  were  conducted.  Results  showed 
that  Elevated  Camera  configuration  was  significantly  faster  than  the  Baseline  configuration, 
/;<().  02.  Results  also  showed  that  significantly  more  time  was  needed  to  complete  the  obstacle 
course  for  the  Waypoint  configuration  than  the  other  three  configurations,  p’s<0.01.  There  were 
no  significant  differences  between  Projected  Path  and  Elevated  Camera  or  between  Projected 
Path  and  Baseline  for  obstacle  course  completion  time. 

Figure  17  presents  the  mean  number  of  obstacle  collisions  for  the  four  CATO  configurations. 
The  percent  change  from  Baseline  (TALON)  is  listed  over  each  bar.  Significant  differences 
(p’s<0.05)  from  Baseline  are  marked  with  an  asterisk.  Elevated  Camera  resulted  in  the  least 
number  of  collisions  and  Waypoint  resulted  in  the  most. 


Note:  *  =  a  significant  difference. 


Figure  17.  Graph  of  mean  (standard  error  of  the  mean)  obstacle  collisions 
for  CATO  configurations. 


A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CATO  configuration  for 
obstacle  collisions,  F  (3,33)  =  8.07,  /;<(). 01 .  There  was  no  significant  effect  in  collisions 
between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  ( 1 , 1 1 )  =  1.96,  p< 0. 1 9 
and  F  (3,33)  =  1.79,  /;<(). 42,  respectively.  To  explain  the  main  effect  of  configuration  on  path 
following  time,  pairwise  comparisons  were  conducted.  Results  showed  that  Elevated  Camera 
and  Projected  Path  configurations  yielded  significantly  less  collisions  than  the  Baseline 
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configuration,  p<0.01  and  /;<().  05,  respectively.  Results  also  showed  that  significantly  more 
collisions  resulted  from  the  Waypoint  configuration  than  the  Elevated  Camera  or  Projected  Path 
configurations,  p’sO.OI .  There  was  no  significant  difference  between  the  Baseline  and 
Waypoint  modes  or  between  Elevated  Camera  and  Projected  Path  for  number  of  collisions. 

Figure  18  presents  the  mean  power  usage  rate  for  the  four  CATO  configurations.  The  percent 
change  from  Baseline  (TALON)  is  listed  over  each  bar.  Significant  differences  (p’s<0.05)  from 
Baseline  are  marked  with  an  asterisk.  Despite  taking  longer  to  complete  both  the  path  following 
and  obstacle  sections  of  the  course,  Waypoint  uses  the  least  power  per  minute  of  operation. 
Elevated  Camera  was  found  to  use  the  most  power  per  minute  of  operation. 


Note:  *  =  a  significant  difference. 


Figure  18.  Graph  of  mean  (standard  error  of  the  mean)  power  consumption  rates  for  CATO 
configurations. 


A  repeated  measures  ANOVA  revealed  a  significant  main  effect  of  CATO  configuration  for  rate 
of  power  consumption,  F  (3,33)  =  16.32,  p<0.01.  There  was  no  significant  effect  in  collisions 
between  trial  1  and  trial  2  or  an  interaction  of  trial  with  configuration,  F  (1,11)  =  3.33,  /;<(). 09 
and  F  (3,33)  =  2. 10,  p< 0. 12,  respectively.  To  explain  the  main  effect  of  configuration  on  path 
following  time,  pairwise  comparisons  were  conducted.  Results  showed  that  the  Waypoint 
configuration  yielded  significantly  less  rate  of  power  consumption  than  the  other  three 
configurations,  p’s<0.03.  Results  also  showed  that  significantly  less  power  per  minute  was  used 
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by  the  Baseline  configuration  than  the  Elevated  Camera  and  Projected  Path,  /;<(). 01  and  /;<(). 02, 
respectively.  There  was  no  significant  difference  between  the  Elevated  Camera  and  Projected 
Path  modes  for  the  rate  of  power  consumption. 

Power  Consumption  is  not  correlated  with  any  individual  metric.  However  there  are  some 
interesting  observations.  First,  although  overall  completion  time  is  similar  for  Baseline,  Elevated 
Camera,  and  Projected  Path,  the  Baseline  mode  uses  significantly  less  power.  Second,  while 
Elevated  Camera  and  Projected  Path  modes  perform  better  in  terms  of  path  precision  and 
avoiding  collisions  than  Baseline,  the  cost  is  higher  power  consumption. 

A  MANOVA  was  conducted  to  examine  the  effects  of  the  four  CATO  configurations  (Baseline  - 
TALON,  Elevated  Camera,  Projected  Path,  and  Waypoint)  on  subjective  workload.  Subsequent 
ANOVAs  and  pairwise  comparisons  were  conducted  for  the  significant  workload  subscales. 

Figure  19  presents  a  graph  of  mean  workload  across  the  CATO  configurations.  Results  showed 
that  subjective  workloads  were  higher  in  the  Waypoint  and  TALON  configurations  than  the 
Elevated  Camera  and  Projected  Path  for  all  six  subscales.  Table  4  lists  the  percent  change  in 
workload  from  Baseline  for  the  three  CATO  modes. 


Figure  19.  Graph  of  mean  (standard  error  of  the  mean)  subjective  workload  for 
CATO  configurations. 
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Table  4.  Table  of  percentages  of  workload  from  baseline  (TALON)  by  configuration. 


Mental 

Physical 

Temporal 

Performance 

Effort 

Frustration 

Overall 

Baseline  (TALON) 

0.0% 

0.0% 

0.0% 

0.0% 

0.0% 

0.0% 

0.0% 

Elevated  Camera 

-10.7% 

-23.5% 

-22.1% 

-22.3%a 

-18.9% 

-35.2%a 

-19.3% 

Projected  Path 

-15.3% 

-29.5%a 

-12.2% 

-22.6% 

-11.6% 

— 38.3%a 

-14.9% 

Waypoint 

39.1%a 

32.4% 

22.9% 

25.9% 

19.5% 

27.2% 

30%a 

d  Significant  comparison  between  configuration  and  baseline  (TALON),  p< 0.05. 


A  repeated  measures  MANOVA  revealed  a  main  effect  of  CATO  configuration  on  workload, 
Wilks  X  (21,  69)  =  2.3 1 ,  /;<().()  1 .  There  was  a  significant  difference  in  workload  between  trial  1 
and  trial  2,  F  (7,4)  =  6.86,  /;<(). 041 .  There  was  not  an  interaction  of  trial  with  configuration, 

F  (21,69)  =  1.06,  p<0.41 .  Subsequent  ANOVAs  were  conducted  to  determine  which  subscales 
of  the  NASA-TLX  workload  scale  contributed  to  this  main  effect.  There  were  significant  main 
effects  of  CATO  configuration  for  all  six  subscales — Mental:  F  (3,30)  =  8.05,  /?<().()  1 ;  Physical: 
F  (3,30)  =  6.44,  p< 0.01;  Temporal:  F  (3,30)  =  4.47,  p<0.01;  Performance:  F  (3,30)  =  7.76, 
p<0.01;  Effort:  F  (3,30)  =  5.65,  p<0.0 1 ;  Fatigue:  F  (3,30)  =  10.38,  pcO.Ol.  To  explain  the  main 
effect  of  configuration  on  the  significant  subscales,  pairwise  comparisons  were  conducted. 
Results  showed  that  there  were  no  significant  differences  between  Elevated  Camera  and 
Projected  Path  for  any  of  the  subscale  ratings.  There  were  no  significant  differences  between 
TALON  and  Waypoint,  except  for  a  significantly  higher  mental  demand  for  Waypoint 
configuration,  /;<(). 02.  Waypoint  yielded  significantly  higher  workload  than  both  Elevated 
Camera  and  Projected  Path  for  all  six  subscales,  p’s  <0.01.  TALON  had  significantly  higher 
workload  than  Elevated  Camera  for  performance  and  frustration,  /;<(). 05  and  p<0.01, 
respectively.  TALON  configuration  had  significantly  higher  workload  than  Projected  Path  for 
physical  and  frustration. 

A  second  repeated  measures  ANOVA  was  conducted  for  overall  workload.  There  was  a 
significant  main  effect  between  CATO  configurations  for  overall  workload,  F  (3,30)  =  9.86, 
p<0.0 1 .  To  explain  the  main  effect  of  configuration  on  the  overall  workload,  pairwise 
comparisons  were  conducted.  Waypoint  had  significantly  greater  overall  workload  than  the 
other  three  configurations,  p’s<0.01.  There  were  no  other  significant  differences  between  the 
configurations  for  total  workload. 
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4.  Conclusions 


The  objective  of  this  research  was  to  experimentally  assess  the  value  added  of  the  CARMAN 
and  CATO  technologies  to  manipulation  and  tele-operation,  respectively.  To  achieve  this  goal, 
ARL-Human  Research  and  Engineering  Directorate  (HRED)  in  collaboration  with  AMRDEC 
developed  an  experiment  plan.  AMRDEC  provided  the  hardware  and  software  necessary  to 
complete  the  experiment.  ARL  conducted  the  experiment  and  analyses  described  in  this  report. 

4.1  CARMAN 

For  CARMAN,  a  light  board  was  built.  Targets  that  varied  in  size  and  location  were  projected 
on  the  board.  Participants  used  the  CARMAN  technologies  to  move  the  manipulator  arm  to  the 
target.  The  CARMAN  technologies  used  were  the  Baseline  configuration  (ALTON),  Fly-To, 
and  Point  and  Click  configuration.  Results  showed  that  performance  with  the  Fly-To 
configuration  was  superior  to  the  other  two  configurations.  More  specifically,  fewer  targets  were 
missed  in  the  Fly-To  configuration  than  either  the  Baseline  (ALTON)  or  the  Point  and  Click 
configuration.  Although  Point  and  Click  was  developed  to  aid  object  manipulation,  it  did  not 
improve  performance  relative  to  the  Baseline  (ALTON).  Interestingly,  task  time  was  longer  for 
Point  and  Click  than  Baseline,  which  was  contrary  to  our  prediction.  In  addition  to  objective 
performance,  we  assessed  operator  workload  after  each  experimental  trial.  The  subjective  data 
complemented  the  objective  results.  In  the  Point  and  Click  mode  participants  reported  more  time 
pressure,  irritation,  and  stress  during  the  task  than  in  the  Baseline  configuration.  The  reason  for 
these  objective  and  subjective  findings  may  be  in  the  implementation  of  Point  and  Click 
configuration.  In  this  mode  the  participant  lined  up  the  manipulator  with  the  target  and  then  the 
arm  moved  close  to  the  location.  The  participant  then  had  to  switch  to  the  basic  mode  and 
complete  the  manipulation  task  manually.  In  contrast,  the  Fly-To  configuration  did  not  require 
this  last-minute  manual  adjustment  by  the  operator.  Thus,  the  Fly-To  technologies  successfully 
improved  object  manipulation,  moving  the  manipulator  arm  to  a  target  of  interest.  This 
technology  improved  the  speed  and  accuracy  with  which  the  operator  reached  a  target,  with 
lower  reported  workload  ratings. 

4.2  CATO 

For  the  CATO  task  an  obstacle  course  was  created  outside.  The  course  included  a  painted  path 
to  follow,  gates,  slaloms,  and  obstacles.  Participants  used  the  CATO  technologies  to  maneuver 
through  the  course.  The  course  was  primarily  divided  into  two  portions;  a  path  following  section 
and  an  obstacle  course  section.  The  CATO  technologies  were  the  Baseline  configuration 
(TALON  IIIB),  Elevated  Camera,  Projected  Path,  and  Waypoint  configuration.  Across  all  tasks, 
Waypoint  configuration  was  found  to  be  the  least  suitable.  This  is  not  surprising  since  the 
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proposed  application  of  Waypoint  configuration  is  to  allow  the  operator  to  move  in  a  straight  line 
at  a  constant  speed  while  performing  a  visual  search  or  other  task.  The  CATO  driving  course 
was  not  designed  to  test  that  type  of  UGV  mission.  While  the  three  remaining  configurations 
allowed  completion  of  the  path-following  section  at  similar  speeds,  the  Projected  Path 
configuration  was  superior  in  minimizing  deviation  from  the  path  to  be  followed.  This  verified 
the  primary  purpose  of  the  Projected  Path  configuration;  to  align  the  robot  with  a  chosen  path.  It 
is  interesting  that  the  operators  did  not  slow  down  when  they  were  having  difficulty  following 
the  path.  The  operators  generally  maintained  a  constant  speed  and  did  their  best  to  apply  a 
“closed-loop”  course  correction  to  the  path  as  they  progressed.  While  both  Projected  Path  and 
Elevated  Camera  showed  improvement  from  the  Baseline  configuration  during  the  obstacle 
course  segment,  the  Elevated  Camera  was  best  at  both  speed  and  accuracy  in  completing  the 
obstacle  course.  The  extra-wide  field  of  view  was  instrumental  in  assisting  the  operator  to  pass 
though  narrow  openings  (such  as  the  slalom  and  gate  obstacles)  in  a  quick  and  efficient  manner. 
The  projected  path  configuration  did  allow  the  operators  to  align  the  robot  with  the  opening,  but 
the  reduction  in  field  of  view  was  a  detriment  to  performance  in  that  configuration.  In  an 
informal  survey,  the  participants  reported  an  overwhelming  preference  for  using  Projected  Path 
for  the  path  following  segment  and  a  nearly  even  split  for  preferring  Projected  Path  or  Elevated 
Camera  for  the  obstacle  course.  This  preference  was  echoed  in  the  decreased  workload  scores 
for  Elevated  Camera  and  Projected  Path  compared  to  Baseline  or  Waypoint  configurations. 

4.3  Summary 

The  objective  of  this  research  was  to  experimentally  evaluate  the  effect  of  CARMAN  and  CATO 
technologies  on  manipulator  activity  and  tele-operation,  respectively.  These  technologies  were 
assessed  to  gauge  their  effectiveness  relative  to  standard  operation  of  a  TALON  TUB. 

CARMAN  involves  the  development  of  technologies  to  reduce  robotic  manipulator  task  times 
and  improve  the  precision  of  robotic  manipulator  placement.  CATO  involves  the  development 
of  technologies  to  reduce  tele-operated  robotic  mobility  task  times  and  improve  the  precision  of 
tele-operated  driving.  The  results  of  this  technical  test  indicate  that  not  only  do  CARMAN  and 
CATO  technologies  increase  accuracy  and  precision  with  respect  to  manipulator  activity  and 
tele-operation,  but  also  significantly  decrease  mission  times. 
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Appendix  A.  Informed  Consent  Form 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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VOLUNTEER  AGREEMENT  AFFIDAVIT: 

ARL-HRED  Local  Adaptation  of  DA  Form  5303-R.  For  use  of  this  form,  see  AR  70-25  or  AR  40-38 


The  proponent  for  this  research  is: 


U.S.  Army  Research  Laboratory 
Human  Research  and  Engineering 
Directorate 

Aberdeen  Proving  Ground,  MD  21005 


Authority: 

Privacy  Act  of  1974,  10  U.S.C.  3013,  [Subject  to  the  authority,  direction,  and  control  of  the 

Secretary  of  Defense  and  subject  to  the  provisions  of  chapter  6  of  this  title,  the  Secretary  of  the 

Army  is  responsible  for,  and  has  the  authority  necessary  to  conduct,  all  affairs  of  the  Department  of 
the  Army,  including  the  following  functions:  (4)  Equipping  (including  research  and  development), 

44  USC  3101  [The  head  of  each  Federal  agency  shall  make  and  preserve  records  containing 
adequate  and  proper  documentation  of  the  organization,  functions,  policies,  decisions,  procedures, 
and  essential  transactions  of  the  agency  and  designed  to  furnish  the  information  necessary  to  protect 
the  legal  and  financial  rights  of  the  Government  and  of  persons  directly  affected  by  the  agency's 
activities] 

Principal  purpose: 

To  document  voluntary  participation  in  the  Research  program. 

Routine  Uses: 

The  SSN  and  home  address  will  be  used  for  identification  and  locating  purposes.  Information 
derived  from  the  project  will  be  used  for  documentation,  adjudication  of  claims,  and  mandatory 
reporting  of  medical  conditions  as  required  by  law.  Information  may  be  furnished  to  Federal,  State, 
and  local  agencies. 

Disclosure: 

The  furnishing  of  your  SSN  and  home  address  is  mandatory  and  necessary  to  provide  identification 
and  to  contact  you  if  future  information  indicates  that  your  health  may  be  adversely  affected. 

Failure  to  provide  the  information  may  preclude  your  voluntary  participation  in  this  data  collection. 

Part  A  •  Volunteer  agreement  affidavit  for  subjects  in  approved  Department  of  Army  research  projects 


Note:  Volunteers  are  authorized  medical  care  for  any  injury  or  disease  that  is  the  direct  result  of  participating  in  this 
project  (under  the  provisions  of  AR  40-38  and  AR  70-25). 


Title  of  Research  Project: 

Technical  Testing  of  the  CARMAN  and  CATO  technologies 

Human  Use  Protocol  Log  # 
Number: 

ARL-20098-08035 

Principal  Investigator: 

Regina  Pomranky 

U.S.  Army  Research  Laboratory 

Human  Factors  Integration  Division 

Fort  Rucker  Field  Element 

Phone:  334-255-2135 

E-Mail:  rpomranky@arl.army.mil 

Associate  Investigator(s) 

Keryl  A.  Cosenzo,  Ph.  D 

U.S.  Army  Research  Laboratory 

Soldier  Performance  Division 

Crew  Station  Branch 

Phone:  410-278-5885 

E-Mail:  kcosenzo@arl.army.mil 

Associate  Investigator(s) 

Brad  Pettijohn 

U.S.  Army  Research  Laboratory 

Human  Factors  Integration  Division 

Fort  Leonard  Wood  Field  Element 

Phone:  573-563-5326 

E-Mail :  brad. pe ttij ohn  @ us . army . mil 

Associate  Investigator(s) 

Andrew  Bodenhamer 

U.S.  Army  Research  Laboratory 

Human  Factors  Integration  Division 

Fort  Leonard  Wood  Field  Element 

Phone:  573-563-6031 

E-Mail: 

andrew.s.bodenhamer@us.army.mil 

Location  of  Research: 

ARMDEC  Huntsville,  AL 

Dates  of  Participation: 

November  2008 
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Part  B  •  To  be  completed  by  the  Principal  Investigator 

Note:  Instruction  for  elements  of  the  informed  consent  provided  as  detailed  explanation  in  accordance  with 
Appendix  C,  AR  40-38  or  AR  70-25. 


Purpose  of  the  Research 


The  purpose  of  this  research  is  to  test  the  performance  effects  of  the  Computer  Aided  Tele- 
operation  (CATO)  and  Computer  Aided  Robotic  Manipulation  ( CARMAN )  technologies.  The 
CATO  technology  is  a  driving  cuds  software.  The  CARMAN  technology  is  a  manipulator  control 
software  package.  We  are  evaluating  the  new  experimental  driving  aid  and  manipulator  control 
software  to  assess  the  performance  effects  of  operating  ground  vehicles  by  remote  control  with 
and  without  these  new  technologies. 


Procedures 

This  experiment  is  divided  into  three  experiments  during  the  day.  In  Experiment  I,  you  will  control  the  TALONIIIB 
manipulator  arm.  The  objective  will  be  to  manipulate  the  manipulator  arm  to  hit  20  targets  that  will  be  displayed  on 
a  light  board  as  quickly  and  accurately  as  possible.  In  Experiment  II,  you  will  remotely  drive  the  TALONIIIB 
through  an  outdoor  obstacle  course.  The  objective  will  be  to  maneuver  the  unmanned  system  through  the  obstacle 
course  as  quickly  and  accurately  as  possible.  In  Experiment  III,  you  will  remotely  drive  a  HMMWV  through  an 
outdoor  obstacle  course.  The  objective  will  be  to  maneuver  the  unmanned  system  through  the  obstacle  course  as 
quickly  and  accurately  as  possible.  Experimental  I  will  take  place  in  an  indoor  climate-controlled  laboratory. 
Experiment  II  will  take  place  outdoors  on  a  mowed  field  on  which  our  obstacle  course  laid  out.  Experiment  III  will 
take  place  outdoors  on  an  airfield  airstrip. 

You  will  complete  Experiment  I,  II,  and  III  in  one  eight  hour  day.  A  30  minute  break  will  be  taken  between 
each  experiment  and  45  minutes  will  be  given  for  lunch.  At  the  start  of  each  experiment  you  will  be  given  an 
overview  of  and  training  on  the  baseline  and  advanced  configurations  for  the  technology.  You  will  then  complete  a 
training  run.  You  will  have  the  opportunity  to  repeat  the  training  run  until  you  are  comfortable  controlling  the  robot. 
Criteria  for  efficient  training  will  be  determined  by  a  Subject  Matter  Expert  on  the  platform  who  will  be  present 
during  the  training  mission.  These  subject  matter  experts  are  the  engineers  of  the  CARMAN  and  CATO  software 
and  will  use  their  knowledge  of  the  system  to  assess  training  effectiveness.  After  training  you  will  complete  the 
experimental  runs.  At  the  end  of  each  run  you  will  complete  the  NASA-TLX  to  assess  you  workload  during  the  task. 
Total  participation  time  will  be  approximately  eight  hours. 

Benefits 

You  will  receive  the  personal  satisfaction  of  providing  valuable  information  to  the  Army's  unmanned 
systems  research. 


Risks 

The  risks  that  will  be  encountered  in  this  study  are  minimal  and  typical  of  the  everyday  risks  encountered  by 
military  and  civilian  personnel  performing  office  duties  using  their  computers.  Mild  motion  sickness  may  be 
experienced  due  to  operation  of  an  operator  control  unit  to  control  an  unmanned  system.  A  motion  sickness 
questionnaire  will  be  used  to  assess  this  risk.  A  comparison  of  the  pre-  and  post-experimental  response  patterns 
across  the  questionnaire  will  be  conducted  to  evaluate  motion  sickness.  If  you  show  elevated  scores  on  any  of  the 
items,  you  will  be  held  at  the  research  site  until  their  symptoms  abate. 
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Photography 


Photographs  and  video  may  be  recorded  during  the  experiment.  Photographic  or  video  images  of  you 
taken  during  this  data  collection  will  not  be  identified  with  any  of  your  personal  information  (name, 
rank,  or  status). 


We  would  like  your  permission  to  take  pictures/videotape/audio  record  during  the  experimental  session.  The 
pictures/recording  may  be  used  in  reports  or  presentations  of  this  work.  Please  indicate  below  if  you  will  agree  to 
allow  us  to  record  you.  You  can  still  be  in  the  study  if  you  prefer  not  to  be  recorded. 


I  give  consent  to  be  audio  taped  d urine  this  studv:  Yes 

_No 

please  initial: 

I  give  consent  to  be  videotaped  during  this  studv:  Yes 

_No 

please  initial: 

I  eive  consent  to  be  photographed  during  this  studv:  Yes 

_No 

please  initial: 

Confidentiality 

All  data  and  information  obtained  about  you  will  be  considered  privileged  and  held  in  confidence.  All  examinations 
will  be  recorded  using  a  volunteer  identifier  code  and  a  separate  file  with  your  consent  form  and  the  Principal 
Investigator  will  keep  your  assigned  volunteer  identifier  code  in  a  locked  cabinet.  Complete  confidentiality  cannot 

be  promised,  particularly  if  you  are  a  military  service  member,  because  information  bearing  on  your  health  may  be 
required  to  be  reported  to  appropriate  medical  or  command  authorities.  In  addition,  applicable  regulations  note  the 
possibility  that  the  U.S.  Army  Human  Research  Protection  Office  officials  may  inspect  the  records.  In  order  to 
ensure  that  your  data  will  not  be  reported  or  revealed  to  anyone,  each  form  will  be  reviewed  upon  receipt  by  one  of 
the  investigators.  If  any  identifying  information  appears  on  the  questionnaires  (such  as  name,  social  security 
number,  birth  date,  etc.),  the  investigators  will  delete  the  identifying  information  and  replace  it  with  a  neutral  code 
number.  All  investigators  perusing  the  questionnaires  and  forms  for  any  sensitive  subject  information  put  on  the 
forms  have  taken  Human  Use  Training. 

Disposition  of  Volunteer  Agreement  Affidavit 

The  Principal  Investigator  will  retain  the  original  signed  Volunteer  Agreement  Affidavit  and  forward  a  photocopy  of 
it  to  the  Chair  of  the  Human  Use  Committee  after  the  data  collection.  The  Principal  Investigator  will  provide  a  copy 
of  the  signed  and  initialed  Affidavit  to  you. 

Contacts  for  Additional  Assistance 


If  you  have  questions  concerning  your  rights  on  research-related  injury,  or  if  you  have  any  complaints  about  your 
treatment  while  participating  in  this  research,  you  can  contact: 


Chair,  Human  Use  Committee 
U.S.  Army  Research  Laboratory 
Human  Research  and  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 

(410)278-5992 


OR  Office  of  the  Chief  Counsel 

U.S.  Army  Research  Laboratory 
2800  Powder  Mill  Road 
Adelphi,  MD  20783-1197 

(301)  394-1070  or  (DSN)  290-1070 
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I  do  hereby  volunteer  to  participate  in  the  research  project  described  in  this  document.  I  have  full  capacity  to  consent 
and  have  attained  my  18th  birthday.  The  implications  of  my  voluntary  participation,  duration,  and  purpose  of  the 
research  project,  the  methods  and  means  by  which  it  is  to  be  conducted,  and  the  inconveniences  and  hazards  that 
may  reasonably  be  expected  have  been  explained  to  me.  I  have  been  given  an  opportunity  to  ask  questions 
concerning  this  research  project.  Any  such  questions  were  answered  to  my  full  and  complete  satisfaction.  Should 
any  further  questions  arise  concerning  my  rights  or  project  related  injury,  I  may  contact  the  ARL-HRED  Human 
Use  Committee  Chairperson  at  Aberdeen  Proving  Ground,  Maryland,  USA  by  telephone  at  410-278-5992  or 
DSN  298-5992. 1  understand  that  any  published  data  will  not  reveal  my  identity.  If  I  choose  not  to  participate,  or 
later  wish  to  withdraw  from  any  portion  of  it,  I  may  do  so  without  penalty.  I  understand  that  military  personnel  are 
not  subject  to  punishment  under  the  Uniform  Code  of  Military  Justice  for  choosing  not  to  take  part  as  human 
volunteers  and  that  no  administrative  sanctions  can  be  given  me  for  choosing  not  to  participate.  I  may  at  any  time 
during  the  course  of  the  project  revoke  my  consent  and  withdraw  without  penalty  or  loss  of  benefits.  However,  I 
may  be  required  (military  volunteer)  or  requested  (civilian  volunteer)  to  undergo  certain  examinations  if,  in  the 
opinion  of  an  attending  physician,  such  examinations  are  necessary  for  my  health  and  well-being. 


Printed  Name  Of  Volunteer  (First,  MI.,  Last) 

Social  Security  Number  (SSN) 

Permanent  Address  Of  Volunteer 

Date  Of  Birth 
(Month,  Day,  Year) 

Today’s  Date 
(Month,  Day,  Year) 

Signature  Of  Volunteer 
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Intentionally  left  blank. 
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Appendix  B.  Demographics  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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Participant  ID# _ 

Demographic  and  Computer  Experience  Questionnaire 

1.  AGE:  _ 

2.  GENDER:  _ Male  _ Female 

3.  Do  you  weai-  glasses? _ Yes _ No 

4.  Do  you  have  any  reason  to  believe  that  you  have  a  hearing  impairment? _ Yes  _ No 

5.  Please  indicate  your  highest  level  of  education: 

_ High  School  Diploma 

_ Undergraduate  Degree 

_ Some  graduate  courses 

_ Graduate  Degree 

_ Other 

6.  Are  you  or  have  you  been  in  the  military?  _ Yes  _ No  If  yes,  what  Branch? 

For  how  many  years?  _ Less  than  5  years  _ 5-10  years _ 1 1-15  years _ 16-20  years _ 20 

years  or  more 

7.  Does  your  job  require  you  to  use  a  computer  on  a  regular  basis?  _ Yes  _ No 

8.  How  long  have  you  been  using  a  computer? 

_ Less  than  1  year  _ 1-3  years _ 4-6  years  _ 7-10  years  _ 10  years  or  more 

9.  How  often  do  you  use  a  computer? 

_ Daily  Weekly  _ Monthly  _ Once  or  twice  a  year 

10.  Do  you  have  a  computer  in  your  house?  _ Yes  _ No 

11.  Do  you  use  the  computer  to  play  games?  _ Yes  _ No 

If  yes,  how  often?  _ Daily  Weekly  _ Monthly  _ Once  or  twice  a  year 

12.  What  is  your  level  of  experience  with  the  operation  of  unmanned  systems? 

No  previous  experience _ 

Some  experience  (1-3  times)  _ 

Substantial  experience  (More  than  3  times) _ 

If  you  have  experience,  with  what  systems  do  you  have  experience? 
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For  what  application(s)  did  you  use  unmanned  systems? 


Intentionally  left  blank. 
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Appendix  C.  NASA  TLX  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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NASA  TLX  Questionnaire 


Participant  ID: 


TLX  Workload  Scale 

Please  rate  your  workload  by  putting  a  mark  on  each  of  the  six  scales  at  the  point  which  matches  your  experience. 


Mental  Demand 


Low 


High 


Physical  Demand  _ [___ _ [ _ j _ _ j__  J_  j _ _[ _ _ j _ ^  _ _  _j _ 

Low  High 


Temporal  Demand  j _  ;  j  j _ ■  j  ,  | _ j _ ' 

Low  High 


Performance  [  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 

Good  Poor 


Effort 


Low 


High 


Frustration 


Low 


High 
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Appendix  D.  Symptom  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  change. 
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Symptom  Questionnaire 


Participant  ID: 


Condition:. _  Date: _  Time: _ 

Using  the  scale  below,  please  rate  how  accurately  the  following  statements  describe  your 
experience. 


1 . 1  feel  sick  to  my  stomach  Not  at  all  Severely 


2. 1  feel  faint-like 

1 - 2— - 

Not  at  all 

1 _ r)—. 

....3.... 

_ 3.... 

....4.... 

_ 4.... 

—5—- 

_ s___. 

-6—- 

_ a.... 

—7 — 

_ 7 _ 

—8—- 

_ 

....9 

Severely 

_ q 

3. 1  feel  annoyed  /  irritated 

1  z. 

Not  at  all 

i _ 7___. 

_ 3.... 

_ A—. 

«J 

_ s.„. 

U 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

4. 1  feel  sweaty 

Not  at  all 

1 _ 0-— 

_ 3.... 

“T 

_ 4_... 

«J 

_ s.„. 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

5. 1  feel  queasy 

Not  at  all 

1 _ 

_ 3.... 

“T 

_ 4_... 

«J 

_ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

6. 1  feel  lightheaded 

1  Z 

Not  at  all 

1 _ 

_ 3.... 

_ 4_... 

_ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

7. 1  feel  drowsy 

Not  at  all 

1 _ 

_ 3.... 

_ A—. 

«J 

_ s___. 

VJ 

_ 6___. 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

A  Z- 

8. 1  feel  clammy  /  cold  sweat  Not  at  all 

1 _ 

_ 3.... 

_ 4_... 

«J 

_ s.._. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

9. 1  feel  disoriented 

A  Z- 

Not  at  all 

1 _ i—. 

_ 3.... 

“T 

_ 4_... 

«J 

_ <5—. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

10. 1  feel  tired  /  fatigued 

1  z- 

Not  at  all 

1 _ 

_ 3.... 

“T 

_ A—. 

«J 

_ s___. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

1 1 . 1  feel  nauseated 

1  Z 

Not  at  all 

1 _ 

_ 3.... 

_ 4_... 

«J 

_ S— . 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

12. 1  feel  hot  /  warm 

1  z- 

Not  at  all 

1 _ 

_ 3.... 

_ 4_... 

«J 

_ s.„. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

13. 1  feel  dizzy 

1  z- 

Not  at  all 

1 _ 

_ 3.... 

“T 

_ A—. 

«J 

_ s.„. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

14. 1  feel  like  I  was  spinning 

1  z- 

Not  at  all 

1 _ 9___. 

_ 3.... 

“T 

_ 4_... 

«J 

_ <5—. 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

15.1  feel  as  if  I  may  vomit 

1  z- 

Not  at  all 

1 _ 

_ 3.... 

“T 

_ 4_... 

«J 

_ 

VJ 

_ 

_ 7 _ 

O 

_ 

y 

Severely 

_ 9 

16. 1  feel  uneasy 

1  Z 

Not  at  all 

1 - 2-- 

....3.... 

....4.... 

«J 

-—5-- 

VJ 

-—6— 

— 7—. 

O 

— 8 — 

y 

Severely 

-—9 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


3-D 

three-dimensional 

AMRDEC 

Aviation  and  Missile  Research,  Development  and  Engineering  Center 

ARL 

U.S.  Army  Research  Laboratory 

BAT 

Battlefield  Automation  Team 

CARMAN 

Computer  Aided  Robotic  Manipulation 

CATO 

Computer  Aided  Tele-operation 

EOD 

explosive  ordnance  disposal 

HRED 

Human  Research  and  Engineering  Directorate 

IED 

improvised  explosive  device 

MTBF 

mean  time  between  failures 

MTTR 

mean  time  to  repair 

NASA-TLX 

NASA-Task  Load  Index 

OCU 

Operator  Control  Units 

SED 

Software  Engineering  Directorate 

SUGV 

small  unmanned  ground  vehicle 

UAV 

unmanned  aerial  vehicle 

UGV 

unmanned  ground  vehicle 
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RDRL  CIM  L 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIM  P 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 


ABERDEEN  PROVING  GROUND 

1  DIR  USARL 

RDRL  CIM  G  (BLDG  4600) 
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NO.  OF 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  A  I  MARTIN 
MYER  CENTER  BLDG  2700  RM  2D31 1 
FORT  MONMOUTH  NJ  07703-5601 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 
320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI  T  DAVIS 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 
FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  R  SPINE 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD 
FORT  BEL  VOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AJ  J  HANSBERGER 
JFCOM  FE 

115  LAKE  VIEW  PKWY  STE  B 
SUFFOLK  VA  23435 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  1 1  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 
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1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE  RM  348 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DW  E  REDDEN 
BLDG  4  CL  60 

FORT  BENNING  GA  31905-5400 

1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  203 1 0-0300 


ABERDEEN  PROVING  GROUND 

5  DIR  USARL 
RDRL  CIM  G 
S  FOPPIANO 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM  B 
J  LOCKETT 
RDRL  HRS  D 
B  AMREIN 


46 


